Skip to content

feat(tracing): stamp sgp evals run and row ids on SGP spans - #545

Open
mohammadatallah-scale wants to merge 10 commits into
nextfrom
mohammad/ove-1253-sgp-evals-span-metadata
Open

mohammadatallah-scale wants to merge 10 commits into
nextfrom
mohammad/ove-1253-sgp-evals-span-metadata

Conversation

@mohammadatallah-scale

@mohammadatallah-scale mohammadatallah-scale commented Oct 7, 2026 •

Copy link
Copy Markdown

OVE-1253. Pairs with an sgp-evaluations PR that tags each generation-unit task.

The problem. Spans of an evals generation-unit task carry no run or row id, so a spans search cannot find them.

The fix.

  • Stamp: the SGP copy of each span of an eval task gets flat run, row and attempt keys. A task that loses the eval marker stops being stamped.
  • Sync and async agents: the ACP server reads the task it receives.
  • Temporal agents: the ids travel through the workflow memo to the worker's activities, child workflows included. A span keeps the ids its task had when it started, so a reused task id cannot strip a queued span. Not covered: spans whose trace_id and task_id are both not the task id.

Test plan

  • Real Temporal dev server run: eval workflow stamps the three keys, a plain one stamps nothing
  • Real Temporal dev server run with a child workflow: its activity is stamped with the parent task's ids (unstamped before this change), and a history recorded with the old interceptor replays with no nondeterminism

Changes since review

  • A span's run and row ids are fixed when it starts, so a task id reused before export cannot change them.
  • A span that started with no eval task can no longer pick up a later run's ids.
  • A failed upload keeps the ids, so a retry sends the same span.
  • Activities of a child workflow carry the ids of the task that started it.

RetriggerView in GreptileConfidence Score: 4/5

This PR is not ready to merge because an early plain span can be saved with a later eval run’s IDs.

Fix All in CursorFindings

  1. P1 Plain spans get eval labels ▶
Fix with agent prompt
### Issue 1
src/agentex/lib/core/tracing/sgp_evals.py:95-96
If a plain span starts before an SGP processor exists, `capture_for_span` saves no empty marker. If an eval task later takes the same task ID, that earlier span can export with the eval task’s run and row IDs. It then appears in searches for work it did not do. Keep the empty result from the span’s start.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Eval-marked tasks now add flat run, row, and attempt ids to their SGP span copies, making spans searchable by evaluation. The PR wires that attribution through ACP handlers and Temporal workers, while keeping queued spans tied to the task metadata they had when they started.

  • SGP span copies show which eval run, row, and attempt they belong to.
  • Temporal workers carry eval ids from workflows into activity spans.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Task[Eval task metadata] --> Memo[Workflow memo]
  Memo --> Header[Activity headers]
  Header --> Registry[Worker task IDs]
  Task --> Registry
  Registry --> Capture[Span-start capture]
  Capture --> Export[SGP span export]
  Registry -->|No capture saved| Export
Loading

Reviews (6) · Last reviewed commit: "style: sort interceptor imports" · Reviewed by Greptile

mohammadatallah-scale and others added 2 commits October 7, 2026 08:04
Spans of a task whose task_metadata carries sgp_evals now get flat
sgp_evals_generation_run_id, sgp_evals_row_id and sgp_evals_attempt_idx keys on
the SGP copy. Sync and async agents read the task in the ACP server. Temporal
agents carry it in the workflow memo and activity headers to the worker.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…val attrs

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
@mohammadatallah-scale

Copy link
Copy Markdown
Author

P3 sgp_evals_interceptor.py - activities of a child workflow get no header (memo is not inherited and their workflow id is not the task id), so their spans stay unstamped even when trace_id is the task id.
P3 worker.py:278 - no test pins SGPEvalsInterceptor in the worker's interceptor list, so dropping it keeps the suite green.

Refuted: continue-as-new keeps the memo (real dev-server run stamps the post-CAN run), ContextInterceptor headers coexist with this one in either order, and task_metadata does reach the agent inside params.task, so the event/send pop never drops a live eval task.

mohammadatallah-scale and others added 2 commits October 7, 2026 08:19
Local activities go through start_local_activity, which the outbound
interceptor did not override, so their spans carried no run or row id.
Also pin the worker's interceptor wiring.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ry none

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
…id cannot strip a queued span

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
Comment thread src/agentex/lib/core/tracing/processors/sgp_tracing_processor.py Outdated
mohammadatallah-scale and others added 2 commits October 7, 2026 08:56
…load succeeds

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Comment thread src/agentex/lib/core/tracing/sgp_evals.py Outdated
…o a span flood cannot evict them

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@mohammadatallah-scale
mohammadatallah-scale requested review from andrewvo-scale and xsfa and removed request for xsfa October 7, 2026 13:08

@andrewvo-scale andrewvo-scale left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The SDK is the right layer, since it is the only place that sees both the task metadata and the span. The keys it reads (generation_run_id, row_id, attempt_idx under the sgp_evals marker) match what sgp-evaluations writes in _unit_task_metadata. With no eval ids, _sgp_metadata returns exactly what base returns, so existing users see no change. All seven threads are resolved with fixes at head. On the first P1 (unregister on later events), keeping the unregister is right, because the server sends the full task, task_metadata included, on every call. Two non-blocking notes inline.

Comment thread src/agentex/lib/core/tracing/sgp_evals.py
Comment thread src/agentex/lib/core/tracing/sgp_evals_interceptor.py
…ns without an SGP processor

Child workflows inherit no memo and their workflow id is not the task id, so
the parent hands both down in the child's memo and activities carry the task id
in a header. Span pinning now starts only once an SGP processor exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mohammadatallah-scale

Copy link
Copy Markdown
Author

P3 sgp_evals_interceptor.py - activities of a child workflow get no header
P3 worker.py:278 - no test pins SGPEvalsInterceptor in the worker's interceptor list

Both closed. Child workflows now inherit the ids and the task id (a789175), and test_worker_runs_with_the_interceptor_ahead_of_agent_interceptors pins the worker's interceptor list.

Comment on lines +95 to +96
if not _capture_enabled:
return

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Plain spans get eval labels

If a plain span starts before an SGP processor exists, capture_for_span saves no empty marker. If an eval task later takes the same task ID, that earlier span can export with the eval task’s run and row IDs. It then appears in searches for work it did not do. Keep the empty result from the span’s start.

Knowledge Base Used: Testing and compatibility

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agentex/lib/core/tracing/sgp_evals.py
Line: 95-96

Comment:
**Plain spans get eval labels**

If a plain span starts before an SGP processor exists, `capture_for_span` saves no empty marker. If an eval task later takes the same task ID, that earlier span can export with the eval task’s run and row IDs. It then appears in searches for work it did not do. Keep the empty result from the span’s start.

**Knowledge Base Used:** [Testing and compatibility](https://app.greptile.com/scale-ai/-/custom-context/knowledge-base/scaleapi/scale-agentex-python/-/docs/testing-and-compatibility.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Fix in Cursor Fix in Claude Code Fix in Codex

@andrewvo-scale

Copy link
Copy Markdown

@mohammadatallah-scale I re-checked a789175/8a170c1. The child-workflow memo + task-id header fix and skipping pins with no SGP processor both look right, and neither runs without eval ids. Two things before merge:

  • CI is red on 040_other_sdks and 020_state_machine (and 010_agent_chat was cancelled), which all passed on 848212a. The assertions are live-model ones ("Did not use tool requests", "Did not find an agent message"), the image pulls retried, and 040_other_sdks isn't Temporal, so this looks like flake rather than this change. Worth a rerun before merging.
  • Greptile's new P1 on sgp_evals.py:96 is real but narrow: it needs the SGP processor to be created after a span starts and that task to gain eval ids later. Fine to leave as a named gap; my approval stands.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants